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ABSTRACT 

A  new  design  for  an  immersive  ground 
control  station  is  presented  that  allows  operators 
to  monitor  and  control  one  or  more  semi- 
autonomous  unmanned  vehicles.  This  new 
ground  station  utilizes  a  virtual  reality 
visualization  of  the  operational  space  and  the 
graphical  representation  of  multiple  real  time 
information  streams  to  create  a  comprehensive 
immersive  environment  designed  to  significantly 
enhance  an  operator’s  situational  awareness  and 
effectiveness.  The  environment  simultaneously 
informs  the  operator  about  the  position  and 
condition  of  the  vehicles  under  his  or  her  control 
while  providing  an  organizing  context  for  the 
available  information  relevant  to  the 
engagement.  This  design  is  applicable  to  a  range 
of  vehicles  including  unmanned  aerial  vehicles 
(UAVs)  and  unmanned  ground  vehicles  (UGVs). 

1.  INTRODUCTION 

The  complexity  and  capability  of  UAVs 
is  expanding  rapidly  and  the  range  of  missions 
they  are  designed  to  support  is  growing. 
Currently,  unmanned  vehicles  specialize  in 
missions  commonly  categorized  as  “the  dull,  the 
dirty,  and  the  dangerous”  [1].  They  have  proven 
to  be  effective  force  multipliers  that  preserve  the 
lives  of  military  personnel.  However,  the  DOD 
roadmap  calls  for  unmanned  vehicles  to  perform 
more  missions  and  to  be  a  larger  part  of  our 
military  force  with  fewer  operators  [1]. 

Significant  technical  challenges  must  be 
overcome  for  unmanned  vehicles’  potential  to  be 
reached.  Many  of  these  challenges  are  human 
interface  issues,  related  to  improving  the  systems 
used  to  monitor  and  control  these  vehicles.  Chief 
among  these  challenges  is  to  develop  new 
mechanisms  to  expand  the  situational  awareness 
of  the  operator  beyond  the  level  provided  by 
current  “soda  straw”  systems  [2]. 

Additionally,  the  number  of  operators 
required  to  control  an  unmanned  vehicle  has  to 
be  reduced.  Current  methods  require  teams  of 


operators  for  one  vehicle.  This  has  to  be  change 
so  that  one  operator  can  control  many. 
Coordinated  advances  in  the  vehicles  and  the 
command  and  control  interfaces  used  to 
supervise  them  are  required  to  accomplish  this 
goal  of  group  control. 

The  technology  exists  to  create  working 
prototypes  for  group  control  interfaces  based  on 
immersive  synthetic  visualization.  Our  work  in 
this  area  is  motivated  by  research  in  two  related 
areas:  the  first  in  joint  battlespace  visualization, 
and  the  second  in  virtual  reality  (VR)  aided 
teleoperation.  The  VRAC  Virtual  Battlespace 
integrates  information  about  tracks,  targets, 
sensors  and  threats  into  an  interactive  VR 
environment.  Our  VR  aided  teleoperation  work 
successfully  combines  vehicle  simulation, 
location  tracking,  and  a  virtual  world 
representation  to  create  a  control  station  that 
provides  superior  situational  awareness  and 
control  in  the  presence  of  signal  lag.  These  two 
works  can  be  combined  to  create  a  prototype 
future  unmanned  vehicle  control  station. 

1.1  The  Virtual  Battlespace 

In  2000,  a  research  team  at  Iowa  State 
University’s  Virtual  Reality  Applications  Center 
(VRAC)  began  work  with  the  Air  Force 
Research  Lab’s  Human  Effectiveness  Directorate 
and  the  Iowa  National  Guard’s  133rd  Air 
Control  Squadron  to  develop  an  immersive  VR 
system  for  distributed  mission  training  called  the 
Virtual  Battlespace.  The  Virtual  Battlespace 
integrates  information  about  tracks,  targets, 
sensors  and  threats  into  an  interactive  virtual 
reality  environment  that  consolidates  the 
available  information  about  the  battlespace  into  a 
single  coherent  picture  that  can  be  viewed  from 
multiple  perspectives  and  scales  [3].  Visualizing 
engagements  in  this  way  can  be  useful  in  a  wide 
variety  of  contexts  including  historical  mission 
review,  mission  planning,  pre-briefing,  post¬ 
briefing  and  live  observation  of  distributed 
mission  training  scenarios. 

Knowledge  gained  from  the 
development  of  the  Virtual  Battlespace 
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contributed  to  the  idea  of  creating  a  cohesive 
virtual  world  representing  the  status  of  real  time 
and  a  priori  information  about  an  engagement 
[4],  Figure  1  shows  the  Virtual  Battlcspace 
environment  displayed  on  a  four-walled  stereo 
projection  system,  the  C4,  at  VRAC. 


Figure  1.  Battlespace  Environment. 


1.2  The  VR-Aided  Teleoperation  System 

In  2002,  the  same  VRAC  research  team 
began  work  on  a  new  teleoperation  control 
system  combining  vehicle  dynamics  simulation, 
position  and  orientation  tracking,  and  a  virtual 
reality  representation  of  the  operational 
environment  to  create  a  vehicle  control  station 
that  provides  superior  situational  awareness  and 
vehicle  control  in  the  presence  of  signal  lag  [5] 
[6].  The  primary  components  of  this  VR  aided 
teleoperation  system  are  shown  in  Figure  2. 


Figure  2.  General  System  Model 


The  user  controls  the  vehicle  from  a 
virtual  environment  displayed  by  the  image 
generator  using  a  vehicle  buck.  The  operator’s 
commands  are  sent  to  a  dynamics  simulation  that 
uses  these  inputs  to  predict  the  dynamic  state  of 


the  virtual  vehicle.  The  dynamic  state  includes 
information  such  as  position,  velocity, 
acceleration  and  heading.  The  state  created  by 
the  dynamics  engine  is  a  simulated  state,  used  to 
both  position  the  virtual  vehicle  and  provide  a 
desired  path  for  the  teleoperated  vehicle. 

The  teleoperated  vehicle  receives  these 
simulated  states  and  synchronizes  them  to 
account  for  the  lag  and  jitter  generated  by  the 
communications  delay.  The  vehicle  uses  these 
synchronized  simulated  states  as  goal  states.  A 
simulation  run  locally  on  the  vehicle  determines 
the  inputs  required  to  get  the  vehicle  to  approach 
the  simulated  state  from  its  current  state. 

Of  course,  to  calculate  these  inputs,  the 
current  state  of  the  vehicle  must  be  determined. 
A  tracking  system  or  observer  provides  this  state 
information.  The  observer  is  responsible  for 
reporting  the  vehicle’s  state  information  to  the 
dynamics  engine  and  the  remote  vehicle.  The 
dynamics  engine  uses  the  reported  vehicle 
position,  corrected  for  lag  and  subsequent 
vehicle  control,  to  predict  the  likely  future 
position  of  the  vehicle.  As  shown  in  Figure  3, 
this  predicted  position  is  depicted  graphically  as 
a  wire-frame  box  surrounding  the  virtual  vehicle 
that  grows  with  the  difference  between  the 
simulated  state  and  the  vehicle’s  projected  state. 


Figure  3.  Wire-frame  Envelope. 


This  wire-frame  envelope  allows  operators  to 
adjust  their  control  to  obtain  higher  fidelity  with 
the  remote  vehicle,  closing  the  loop  between  the 
operator  and  the  vehicle. 

1.3  Prototype  Test  Results 

In  order  to  test  the  effectiveness  of  the 
VR-Aided  teleoperation  system,  our  research 
team  developed  a  prototype.  In  this  test,  the 
teleoperated  vehicle  was  a  remote  controlled 
model  tank.  Our  research  team  developed  a 
custom  circuit  to  allow  the  tank  to  be  computer 


controlled.  The  tank’s  response  to  controls  was 
measured  to  create  a  computer  simulation  of  the 
tank’s  dynamics  and  response  to  inputs.  Once 
our  team  honed  this  simulation  model,  the  tank’s 
response  to  inputs  could  be  closely  predicted. 
The  computer  running  this  simulation  (the 
dynamics  engine)  was  a  Dell  PC  attached  to  a 
Microsoft  Sidewinder  steering  wheel  set.  The 
dynamics  engine  used  the  tank  simulation  to 
generate  the  simulated  states  (shown  in  Figure  2) 
and  then  sent  those  states  to  the  laptop 
communicating  with  the  RC  tank. 

Our  team  implemented  an  optical 
tracking  system  to  act  as  the  observer.  A  red 
cardboard  square  was  placed  on  the  top  of  the 
tank  towards  its  rear  and  a  blue  square  towards 
its  front.  A  webcam,  situated  at  a  fixed  location 
above  the  operational  environment,  produced  a 
video  stream.  An  image  processing  algorithm 
was  implemented  to  find  the  blue  and  red 
squares  in  the  scene.  Calibration  of  the  camera 
enabled  conversion  of  the  vehicle’s  marker  pixel 
location  into  the  corresponding  location  in  the 
operational  environment.  Incorporating  the  fixed 
distance  between  the  vehicle  markers  and  the 
center  of  the  tank,  the  system  could  determine 
both  the  tank’s  real-world  position  and  its 
heading.  Further,  by  keeping  track  of  the 
previous  position  and  orientation  of  the  tank,  the 
system  could  provide  a  first  order  approximation 
for  the  vehicle’s  linear  and  angular  velocity. 
This  information  comprised  the  real  vehicle  state 
required  by  the  vehicle  and  the  dynamics  engine 
(shown  in  Figure  2).  The  observer  was  run  by  a 
laptop  and  it  communicated  the  vehicle  state 
information  via  standard  network  protocols. 

The  image  generator  (shown  in  Figure 
2)  was  an  SGI  RealityEngine2.  It  received 
simulated  and  real  vehicle  states  from  the  PC  and 
laptop  respectively  and  generated  the  virtual 
world  (shown  in  Figure  3).  VRAC’s  C6  device 
displayed  the  virtual  world  in  a  10  foot  by  10 
foot  room  where  each  wall  is  capable  of 
displaying  a  rear  projected  stereo  image.  In  this 
way,  the  system  immersed  the  user  with  3D 
stereo  graphics  in  every  viewing  direction. 
Inside  the  C6,  our  team  placed  a  steering  wheel 
and  chair.  From  this  position  the  operator 
controlled  the  vehicle  in  the  virtual  environment. 

All  of  the  components  of  the  test  system 
were  connected  on  the  same  low  latency 
network.  However,  this  does  not  reflect  a  typical 
operating  environment  where  there  is  a 


significant  signal  delay  between  the  vehicle  and 
the  operator.  To  simulate  this  crucial  effect, 
each  command  sent  by  the  operator  to  the 
vehicle  was  delayed  before  being  transmitted. 
Likewise,  information  returning  to  the  simulation 
from  the  vehicle  observer  was  delayed.  In  this 
way,  simulated  signal  delay  was  introduced  into 
the  system.  Of  course  constant  signal  delay  is 
not  sufficient  to  model  real  world  behaviors.  To 
simulate  variable  delay,  random  perturbations  in 
the  delay  times  were  introduced  by  fluctuating 
±10%  around  the  input  median  value. 

To  manage  the  changing  signal  delay 
times,  operator  commands  were  buffered  at  the 
vehicle,  to  ensure  that  they  could  be  properly 
spaced  in  time.  Buffering  is  a  common  technique 
used  in  multimedia  systems  to  ensure  smooth 
playback.  Client-based  players  of  streaming 
media  on  the  internet  typically  buffer  a  portion 
of  the  song  or  video  before  it  is  played  so  that 
the  next  frame  is  available  in  time  despite 
unpredictable  signal  delay.  Smooth  playback  of 
the  commands  allows  the  remote  vehicle’s 
onboard  computer  to  more  accurately  follow  the 
simulated  path  that  the  operator  generated. 

To  test  the  system,  the  tank  was  piloted 
through  a  course  of  cone  gates  within  the 
operational  environment  using  three  methods: 
direct  control,  camera-aided  teleoperation  and 
virtual  teleoperation.  For  each  method,  the 
average  time  to  complete  the  driving  task  and  the 
number  of  gates  successfully  navigated  were 
recorded.  Direct  control  provided  the  baseline 
for  vehicle  control  because  it  is  in  some  sense 
optimal;  there  is  no  signal  delay  and  the  operator 
can  see  the  vehicle  directly  within  its  operational 
environment.  Camera-aided  teleoperation 
provided  an  important  benchmark  because  it 
represents  the  most  common  current  interface  for 
teleoperated  vehicles.  Test  runs  were  performed 
by  the  authors  for  all  three  control  methods  with 
three  levels  (one,  five,  and  ten  seconds)  of 
nominal  artificial  signal  delay.  Table  1  shows  the 
tests  that  were  run.  Our  team  measured  the 
average  time  to  completion  and  the  number  of 
cone  gates  successfully  navigated.  While  our 
team  did  not  run  enough  tests  for  a  rigorous 
statistical  analysis,  initial  results  are  promising. 

These  preliminary  results  reveal  that  the 
VR-aided  teleoperation  system  greatly  improved 
operator  performance  when  compared  to  a 
lagged  video-based  teleoperation  system.  With 
VR-aided  teleoperation,  the  average  time  to 


completion  was  not  noticeably  affected  by  signal 
delays  up  to  10  seconds.  In  contrast,  the  camera 
aided  teleoperation  system  completion  times 
increased  rapidly  with  only  a  modest  increase  in 
signal  delay.  Furthermore,  the  situational 
awareness  of  the  operator  was  enhanced  as 
evidenced  by  the  fact  that  fewer  cones  were 
knocked  down  with  VR-aided  teleoperation. 


Table  1.  Test  Results 


Test 

Signal 
Delay  (s) 

Average 
Time  (s) 

Average  Cones 
Navigated 

Direct 

0 

26.0 

5.00 

Camera 

1 

101.1 

4.67 

Camera 

5 

357.7 

4.33 

Camera 

10 

583.5 

4.33 

VR 

1 

32.5 

4.67 

VR 

5 

34.7 

5.00 

VR 

10 

31.0 

4.67 

2.  FUTURE  UAV/UGV  GROUND  STATION 

A  next  generation  unmanned  vehicle 
(UV)  control  station  must  fill  multiple  roles. 
The  system  must  provide  a  comprehensive  view 
of  the  overall  mission.  It  must  clearly  depict  the 
mission  context,  including  all  the  relevant 
geographic  and  political  features  of  the  area  in 
which  the  unmanned  vehicles  are  being  operated. 
Further,  it  must  show  the  position  of  all  the  UVs 
in  the  area.  If  a  group  of  vehicles  is  acting  in 
concert  as  a  swarm,  the  intent  of  the  swarm  must 
be  clear.  This  same  system  must  also  allow  an 
operator  to  understand  the  detailed  status  of  an 
individual  UV  and  afford  rapid  access  to  the 
information  necessary  to  make  specific,  decisive 
and  informed  decisions  about  its  behavior.  This 
means  that  whatever  information  sources  are 
used  in  the  control  of  a  single  UV  must  also  be 
present  in  a  system  that  controls  many. 

A  system  that  employs  a  virtual 
environment  as  the  information  focus  is  an 
excellent  candidate  to  solve  these  challenges.  A 
large  scale  VR  display  is  more  versatile  than 
current  2D  desktop  interfaces  because  it  can  fuse 
information  from  several  sources  into  one 
comprehensive  picture  due  in  part  to  its 
increased  display  area.  This  frees  the  user  from 
having  to  mentally  integrate  information 
displayed  in  different  locations.  A  virtual  world 
can  create  an  integrated  view  of  the  mission  by 
combining  views  from  onboard  cameras  with  the 
radar-derived  positions  of  enemy,  friendly,  and 
unidentified  tracks.  In  contrast,  current  UV 


control  stations  present  all  information  feeds  - 
on-board  cameras,  radar,  sensor  data  and  time 
history  radar  (tracks)  -  in  separate  displays.  The 
operator  must  take  on  the  integration  task, 
significantly  increasing  his  or  her  cognitive  load. 

Our  team  defines  VR  control  as  a 
system  that  uses  a  virtual  world  as  the  UV 
control  interface.  A  requirement  for  VR  control 
is  to  generate  virtual  terrain  that  can  provide 
meaningful  context  for  a  given  mission.  This 
terrain  model  can  be  a  composite  that  fuses 
satellite  imagery,  political  boundary  maps, 
DTED  data  and  other  sources.  The  obstacles  to 
creating  useful  terrain  models  are  not 
unreasonably  high  for  UAVs.  Since  only  general 
landmark  features  are  necessary  to  fly  a  UAV, 
the  terrain  model  need  not  match  every  contour 
of  the  real  terrain  to  be  effective.  As  a  result, 
virtual  terrain  for  VR  control  can  be 
implemented  for  current  generation  UAVs. 

VR  control  is  harder  for  UGVs  than 
UAVs.  The  main  reason  is  that  the  obstacles  to 
creating  useful  terrain  models  are  higher  for 
UGVs.  To  facilitate  ground  control,  a  terrain 
model  must  incorporate  higher  frequency  detail 
to  accurately  simulate  the  motion  of  a  UGV. 
Real  time  onboard  terrain  generation  using  laser 
range  finders  might  be  needed  for  VR  control  to 
be  effective  for  UGVs.  Flowever,  future  UGVs 
are  likely  to  be  semi-autonomous  with  advanced 
sensors,  requiring  only  that  the  operator  lay  out  a 
path  for  the  vehicle  instead  of  driving  it.  Under 
these  conditions,  the  terrain  can  be  less  exact, 
making  UGV  VR  control  easier  to  implement. 

With  VR  control  we  expect  the  operator 
to  feel  a  greater  sense  of  presence  and  to  have  a 
more  complete  understanding  of  a  UV’s  situation 
than  is  possible  with  traditional  2D  desktop 
displays.  The  virtual  world  can  be  as  accurate  a 
representation  of  the  real  world  as  current 
desktop  displays  are  now.  Radar  feeds  can  be 
used  to  position  units,  GPS  can  be  used  to 
pinpoint  UVs  in  the  swarm,  weather  data  from 
balloons  and  satellites  can  be  used  to  generate 
the  weather  in  the  virtual  world,  and  camera 
feeds  can  be  used  to  update  visuals  of  the  targets. 
The  difference  between  current  systems  and  VR 
control  is  that  all  of  this  information  can  be 
integrated  into  a  comprehensive  view,  rather  than 
displayed  in  a  fragmented  way. 

Signal  lag,  a  problem  in  conventional 
video-based  UV  control  systems,  also 
complicates  VR  control  because  each  piece  of 


information  required  to  accurately  represent  the 
elements  in  the  virtual  world  is  affected  by  lag. 
Signal  lag  can  be  critical  in  the  heat  of  combat 
and  in  accurately  hitting  targets  [7].  This  lag 
effect  is  not  unique  to  VR  control  —  lag  is  always 
be  present  in  any  UV  control  system.  However, 
using  VR  control  provides  a  key  advantage  in 
minimizing  the  effects  of  lag.  The  VR 
environment  provides  us  with  the  ability  to 
modify  the  virtual  UV’s  positions  in  the  virtual 
world  dynamically  using  informed  or  dead 
reckoning  [5],  Reckoning  a  position  provides  an 
estimate  of  a  vehicle’s  true,  unlagged  position. 
If  the  display  does  not  use  a  reckoned  position 
for  the  vehicle,  the  operator  is  essentially 
attempting  to  operate  the  UV  “in  the  past”. 
Reckoning  vehicle  positions  can  help  to  alleviate 
the  effects  of  lag  by  allowing  the  operator  to 
control  the  UV  not  in  its  past,  but  in  its  likely, 
but  unknowable,  present  state  [5], 

3.  VR  CONTROL  PROTOTYPE 

We  have  developed  a  prototype  for  VR 
control  of  UVs  based  on  the  Virtual  Battlespace. 
The  prototype  has  two  main  modes,  far  scale  and 
close  scale.  It  can  also  direct  operator  attention 
with  alerts  and  place  video  feeds  “in  situ”  within 
the  virtual  world  to  provide  organizing  context. 

3.1  Far  Scale 

Far  scale  allows  an  operator  to  survey 
the  overall  mission  from  a  strategic  viewpoint 
above  the  engagement.  In  this  mode,  the  virtual 
world  is  used  to  display  most  or  all  of  the  units 
involved,  both  friendly  and  enemy.  The  operator 
uses  this  view  to  get  a  sense  for  the  overall 
mission.  This  view  is  crucial  for  managing 
multiple  UVs  because  it  provides  an  organizing 
context  for  the  engagement  and  displays  spatial 
relationships  between  UV  swarm  elements. 

In  far  scale,  the  operator  is  situated 
thousands  of  feet  in  the  air  with  a  comprehensive 
view  of  the  battlefield  and  its  participants. 
Individual  units  can  be  aggregated,  if 
appropriate,  into  squads  to  simplify  the  visual 
clutter  by  representing  them  as  single  entities. 
Figure  4  shows  the  view  of  a  battle  over  a 
simulation  of  the  test  range  at  Nellis  AFB  in 
Nevada  in  far  scale;  the  red  and  blue  wedges 
represent  aggregated  aircraft.  Colors  represent 
the  allegiance  of  the  units,  with  red  representing 
hostile  and  blue  representing  friendly  units. 


Figure  4.  Far  scale. 

Figure  4  shows  other  valuable 
information  that  the  operator  has  access  to  in  far 
scale.  The  green  wedge  in  front  of  the  blue 
aggregate  unit  in  the  middle  of  the  screen 
represents  the  extents  of  the  squad’s  radar  sweep. 
With  this  information,  the  operator  knows 
exactly  which  units  that  squadron’s  radar  can 
detect.  This  knowledge  is  useful  in  determining 
when  a  squadron  can  be  expected  to  acquire  its 
target. 

The  units  in  far  scale  can  leave  trails 
behind  them,  represented  in  the  virtual  world  by 
a  red  dashed  line.  The  trails  tell  the  operator 
where  units  have  been  and  provide  a  context  for 
tracking  overall  movement.  Potential  SAM 
threats  are  represented  by  pink  wire  frame 
hemispheres  (located  at  the  top  far  edge  of  the 
green  radar  sweep  wedge).  Aircraft  which  fly 
within  this  zone  are  in  danger  of  being  targeted 
and  shot  down;  ground  forces  in  this  zone  must 
contend  with  the  SAM  ground  protection  forces. 

Other  sources  of  information  in  the  far 
scale  can  be  displayed  on  the  two-dimensional 
billboard  located  at  the  top  of  the  screen  in 
Figure  4.  The  billboard  contains  a  traditional 
radar  display,  a  compass,  a  map  showing  current 
geographical  location,  and  a  speed  key.  This 
speed  key  can  be  used  to  determine  the 
approximate  speed  of  a  unit  by  its  color.  The 
other  very  important  piece  of  information  this 
view  provides  is  the  relative  spatial  relationship 
between  terrain,  units  and  threats.  It  is  this  facet 
of  the  virtual  world  that  allows  it  to  provide 
superior  situational  awareness. 

3.2  Close  Scale 


The  system’s  second  mode,  close  scale, 
allows  an  operator  to  view  a  mission  from  the 
perspective  of  a  single  unit.  Close  scale  provides 
a  more  detailed  view  of  a  portion  of  the  mission 
space.  Many  times  an  operator  would  like  to  get 
a  more  complete  view  of  what  is  occurring  near  a 
particular  unit.  This  is  especially  true  for  tasks 
like  target  confirmation  or  direct  control.  For 
these  tasks,  the  operator  switches  to  close  scale 
mode,  allowing  him  or  her  to  follow  closely 
behind  a  particular  unit.  Realism  is  enhanced  in 
close  scale  by  representing  the  vehicle  with 
detailed  unit  models.  Figure  5  shows  close  scale. 


Figure  5.  Close  scale. 


Note  that  in  Figure  5  the  aircraft  of 
interest,  an  F-16,  is  flying  through  a  threat  dome 
of  a  SAM  site.  The  environment  provides 
information  to  the  operator  about  immediate 
danger  to  the  aircraft  as  well  as  how  far  that 
danger  extends.  Aircraft  specific  information  is 
displayed  on  a  simulated  heads  up  display 
(F1UD)  showing  heading,  altitude  and  speed. 
Another  graphical  feature,  available  in  both 
scales  but  more  critical  in  close  scale,  is  height 
sticks.  These  are  poles  attached  to  aircraft  that 
extend  vertically  downward  to  the  ground  and 
are  striped  to  give  a  quick  estimate  of  altitude. 
They  are  highly  visible  and  can  cue  the  operator 
to  the  presence  of  other  aircraft  in  the  area  that 
would  otherwise  be  undetectable  at  this  scale. 
Ground  vehicles  are  also  marked  with  a  stick  to 
make  their  locations  visible. 

If  the  vehicle  of  interest  is  a  UV,  the 
operator  could  use  this  scale  to  control  the 
vehicle  directly,  using  the  VR-aided 
teleoperation  control  model  described  in  the  tank 
simulation  above.  An  operating  envelope  similar 
to  the  one  shown  in  Figure  3  could  be  integrated 
in  the  virtual  world  to  provide  the  operator  with 
feedback  about  uncertainty  in  the  current 
position  of  the  vehicle.  With  this  control 


capability,  an  operator  would  be  able  to  rescue  a 
UV  from  a  situation  that  its  own  control  system 
was  unable  to  handle.  A  key  question  for  us  is 
how  effective  a  single  operator  can  be  at 
managing  the  state  of  several  UAVs  in  a  swarm 
using  these  two  scales.  A  partial  answer  to  the 
question  is  provided  by  alerts. 

3.3  Alerts 

If  one  of  the  UAVs  in  the  swarm 
requires  attention  or  intervention,  it  must  alert 
the  operator.  An  alert  attracts  the  operator’s 
attention,  by  raising  a  cue  either  graphically, 
orally,  haptically,  or  in  combination.  In  response 
to  an  alert,  the  operator  can  switch  between 
scales  manually  or  the  system  could  be 
configured  to  transition  automatically.  Figure  6 
shows  what  an  alert  might  look  like  in  far  scale. 


Figure  6.  UAV  alert  in  far  scale. 


An  alert  in  close  scale  could  be  a 
flashing  arrow  pointing  in  the  direction  of  the 
UAV  in  need.  We  are  exploring  several 
possible  alerts.  For  example,  if  an  alert  happens 
while  the  operator  is  attached  to  a  different  unit 
in  close  scale,  the  application  could  seamlessly 
switch  to  far  scale  to  give  the  operator  a  global 
view  of  the  battlefield.  Alternatively,  the  system 
could  seamlessly  move  the  operator  from  his 
current  focus  of  control  to  the  unit  that  raised  the 
alert.  The  key  concept  in  how  alerts  and  the  two 
scales  work  together  to  allow  swarm  control  is 
that  the  far  view  provides  an  overall  but  less 
specific  view  while  close  scale  provides  a  narrow 
but  highly  focused  view.  The  alert  system  helps 
guide  the  operator’s  attention  to  the  parts  of  the 
engagement  in  which  it  is  most  needed.  The  two 
scales  in  concert  with  alerts  create  a  control 
environment  conducive  to  distributed  command 
of  semi-autonomous  vehicles.  We  believe  that 


the  environment  will  help  alleviate  the  losses  in 
situational  awareness  that  cause  human  operators 
to  make  mistakes. 

3.4  “In  Situ”  Video 

One  of  the  most  important  features  of  a 
UV  control  station  is  the  capability  to  view  the 
UV’s  video  feeds.  As  well  as  being  today’s 
primary  basis  for  vehicle  control,  in  the  case  of 
reconnaissance  missions,  real  time  video  is  the 
most  important  information  the  UAV  gathers. 
Today’s  systems  display  UAV  video  feeds  on 
separate  monitors,  or  within  individual  windows 
in  a  desktop  environment.  This  interface  offers 
limited  contextual  information  for  the  video  and 
as  a  result,  the  operator  must  mentally  position 
the  video  within  the  mission  context,  augmenting 
the  on-board  camera’s  limited  field  of  view. 
This  limited  view  contributes  significantly  to  the 
loss  of  situational  awareness  [8]. 

In  the  VR  based  system,  video  is  still  an 
important  information  source.  In  its  simplest 
form,  video  from  a  UV  can  be  played  on  a 
billboard  display,  as  shown  in  Figure  7. 


Figure  7.  Video  feed  in  a  fixed  location. 


While  playing  video  on  a  billboard 
within  the  environment  does  fuse  the  virtual 
world  with  the  real  time  sensor  feed,  it  is  similar 
to  current  display  systems  in  that  it  requires  the 
operator  to  provide  the  video’s  context.  An 
interesting  alternative  is  to  place  the  video  “in 
situ”,  fusing  the  video  image  in  place  with  the 
rest  of  the  virtual  world  to  provide  the  missing 
contextual  information.  In  this  mode,  video 
from  a  UV  feed  is  superimposed  in  its  correct 
position  within  the  virtual  environment,  either  by 
texturing  the  terrain,  in  the  case  of  a  ground 
directed  camera,  or  on  a  “suspended  screen”  in 
the  case  of  an  on  board  camera.  With  “in  situ” 
placement,  an  operator  can  easily  place  video 
information  in  context.  Furthermore,  contrast 


between  the  terrain  shown  and  the  virtual  terrain 
it  replaces  can  provide  information  about  how 
accurately  the  virtual  terrain  matches  the  actual 
terrain.  “In  situ”  video  placement  in  the  virtual 
world  allows  the  operator  to  confirm  the  vantage 
point  that  the  video  is  taken  from,  rather  than 
having  to  infer  it  from  pre-briefing  information 
or  by  integrating  information  from  other 
displays.  Figure  8  shows  an  example  of  an  “in 
situ”  video  taken  from  a  UAV. 


Figure  8.  Video  feed  in  a  fixed  location. 


A  further  method  to  introduce  video 
into  the  scene  is  to  lay  the  images  coming  from  a 
UAV  along  the  vehicle’s  flight  path  while  in  far 
scale.  Using  this  method,  the  video  stream  can 
be  used  to  generate  a  dynamic  ground  texture 
that  contains  the  latest  information  available 
from  the  UAV  camera.  For  example,  if  an 
operator  wished  to  look  at  what  the  UAV  “saw” 
when  it  passed  over  a  particular  mountain  range, 
he  or  she  could  navigate  to  that  mountain  range 
in  the  virtual  world  and  select  the  video  strip  on 
the  UAV’s  flight  path  at  that  location. 

NASA  pioneered  path  specific  video 
placement  with  their  virtual  environment  vehicle 
interface  (VEVI)  for  the  Mars  Pathfinder 
Mission  [9].  The  VEVI  system  used  these 
videos  for  review  of  mission  data,  not  control  of 
the  vehicle.  VEVI  is  shown  in  Figure  9. 


Figure  9.  VEVI  Interface. 

Each  of  the  video  clips  in  Figure  9  is 
displayed  over  the  point  on  the  Martian 


landscape  where  the  clip  was  taken.  If  a  scientist 
wants  to  review  what  the  Pathfinder  saw  at  a 
certain  location,  he  simply  selects  the  video  clip 
closest  to  that  location  and  plays  it.  VEVI 
demonstrated  that  “the  ability  to  continually  see 
all  around  the  robot  provided  scientists  with  a 
more  natural  sense  of  position  and  orientation  . . . 
than  is  usually  available  through  more  traditional 
imaging  systems”  [10].  The  authors  also  note 
that  “this  capability  ...  substantially  accelerated 
site  exploration”  [10].  We  are  apllying  this  same 
“in  place”  philosophy  to  UV  control  to  allow  the 
operator  to  quickly  locate  and  review  footage 
from  the  UV’s  camera,  either  from  the  past  or  in 
real  time. 

In  order  for  a  single  operator  to 
successfully  control  multiple  UVs  he  must 
constantly  juggle  his  attention  from  one  UV  to 
another,  have  all  the  needed  information 
displayed  in  one  location,  and  understand  the 
relationships  between  the  UVs  under  their 
control,  as  well  as  potential  targets  and  threats. 
A  virtual  environment  that  uses  camera  feeds  to 
augment  the  virtual  battlefield  can  accomplish  all 
of  these  tasks  in  a  way  that  is  flexible  and 
extensible.  Since  most  of  the  complexity  is  in 
software,  new  ways  to  represent  the  information 
can  be  implemented  quickly  to  respond  to  new 
technology  and  user  requirements. 

4.  CONCLUSION 

The  Department  of  Defense  has 
allocated  $2.8  million  in  the  2005  defense 
appropriations  bill  to  fund  research  in  virtual 
teleoperation  for  unmanned  vehicles  at  VRAC. 
The  goal  of  this  research  effort  is  to  develop  VR 
based  technologies  to  create  control  and 
monitoring  interfaces  that  work  toward  the  DOD 
Roadmap's  goals  for  simplifying  the  command 
and  control  of  groups  of  unmanned  vehicles  in  a 
variety  of  missions  [1].  The  core  of  our 
approach  is  a  synthetic  a  priori  virtual  model  of 
a  mission  space  that  incorporates  dynamic 
elements  whose  representations  within  the 
virtual  space  are  driven  by  real-time,  or  near 
real-time,  sensor  feeds.  Our  goal  is  to  identify 
and  solve  the  problems  associated  with  using  this 
virtual  environment  to  monitor  and  operate 
unmanned  vehicles  in  a  variety  of  missions  and 
identify  the  appropriate  connections  between  this 
research  and  other  research  in  the  unmanned 
community. 

The  research  group  at  VRAC  is  actively 
seeking  industry  and  defense  agency 


stakeholders  who  can  help  guide  and  focus  this 
effort  and  help  identify  the  most  crucial  points  of 
integration.  In  keeping  with  Iowa  State 
University’s  land  grant  mission,  we  maintain  a 
strong  commitment  to  identifying  mechanisms 
for  effective  technology  transfer  of  results  of  this 
research  to  make  a  positive  impact  on  the 
operation  of  these  vehicles  in  the  real  world. 
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